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Abstract. Let A be a smooth projective variety over C. We find the natural 
notion of semistable orthogonal bundle and construct the moduli space, which we 
compactify by considering also orthogonal sheaves, i.e. pairs (E, ip), where E is a 
torsion free sheaf on X and ip is a symmetric nondegenerate (in the open set where 
E is locally free) bilinear form on E. We also consider special orthogonal sheaves, 
by adding a trivialization ip of the determinant of E such that det(<p) = if) 2 ; 
and symplectic sheaves, by considering a form which is skewsymmetric. More 
generally, we consider semistable tensors, i.e. multilinear forms on a torsion free 
sheaf, and construct their projective moduli space using GIT. 

Let X be a smooth projective variety of dimension n over C. If X is a curve, 
the moduli space of vector bundles was constructed by Mumford, Narasimhan and 
Seshadri. If dim(X) > 1, to obtain a projective moduli space we have to consider 
also torsion free sheaves, and this was done by Gieseker, Maruyama and Simpson. 

An orthogonal bundle is a pair (E, <p), where E is a vector bundle and 

if : E ® E — ► Ox 

is a bilinear symmetric nowhere degenerate form. The nondegeneracy means that 
it induces an isomorphism E — > E v , hence (det E) 2 = Ox- 

A special orthogonal bundle is a triple (E, ip, ip) where E and tp are as before, and 

ip : det E — ► O x 

is an isomorphism such that det (tp) = ip 2 (this equation means that for all points 
x € X, if we choose a basis for the fiber E x , the determinant of the matrix associated 
to <f at x is equal to the square of the scalar associated to ip at x). 

A symplectic bundle is a pair {E, ip), where E is a vector bundle and ip is a bilinear 
skewsymmetric nowhere degenerate form. 

Note that giving an orthogonal (or special orthogonal, or symplectic) bundle 
is equivalent to giving a principal bundle with group structure group 0(r, C) (or 
SO(r, C), or Sp(r, C)). To obtain a projective moduli space we have to consider 
also orthogonal, or special orthogonal or symplectic sheaves, i.e. allowing E to be 
a torsion free sheaf, and then requiring <p to be nondegenerate only on the open 
subset of X where E is locally free. We say that a subsheaf F of E is isotropic if 
v\f®f = 0. 

An orthogonal, or special orthogonal or symplectic sheaf is called stable (respec- 
tively semistable) if for all proper isotropic subsheaves F of E 

P F + P F ± -< P E (respectively X), 

where Pp is the Hilbert polynomial of F, F^ is the sheaf perpendicular to F with 
respect to the form ip, and, as usual, the inequality between polynomials Pi ~< P2 
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(respectively ■<) means that Pi(m) < P2(m) (respectively <) for m ^> (see sections 

5 and 6 for precise definitions). 

A similar problem was considered by Sorger [So]. He works on a curve C (not 
necessarily smooth) on a smooth surface S, and constructs the moduli space of 
torsion free sheaves on C together with a symmetric form taking values on the 
dualizing sheaf u>c- Faltings [Fa] has considered principal bundles on semistable 
curves. For G orthogonal or symplectic he considers a torsion free sheaf with a 
quadratic form, and he also defines a notion of stability. For general reductive 
group G he uses the approach of loop groups. 

More generally, we can consider triples (E, tp, u) where E is a torsion free sheaf 
on X and <p is a non-zero homomorphism 

,p : (£®*)ec — > (det E)® b ®D u , 

where D u is a locally free sheaf belonging to a fixed family {D u } u£ r parametrized 
by a scheme R (for instance, R could be Pic a (A), and then D u is any line bundle 
with fixed degree a, or we can take R to be a point, and then D u is a fixed locally 
free sheaf). We call these triples tensors. See section 1 for the precise definition. 
Tensors generalize several objects that have already appeared in the literature. If 
s = 1, b = 0, c = 1, and R is a point, these are the framed modules of Huybrechts and 
Lehn. They gave two constructions of their moduli space: In [H-Ll] for dim(X) < 2, 
based in the ideas of Gieseker [Gi], and in [H-L2] for arbitrary dimension, following 
the ideas of Simpson [Si]. If dim(X) = 1, s = 2, b = 0, c = 1, R is a point and 
D u is a line bundle, these are the conic bundles of [G-S]. If dim(X) = 1 and V is a 
family of line bundles, these are the decorated vector bundles whose moduli space 
was constructed by Schmitt [Sch]. 

Using geometric invariant theory (GIT) as in [Si] and [H-L2], we construct the 
moduli space of semistable tensors (sections 1 to 4). This is used in sections 5 and 

6 to construct the projective moduli space of classical sheaves. 

Finally, in section 7, as a further application we obtain moduli spaces for GL(r, C)- 
representational pairs, i.e. pairs (P, a) consisting of a principal GL(r, C)-bundle P 
and a section a of the vector bundle associated to a fixed representation of GL(r, C). 
We can also consider a quasi-projective scheme Y with an action of GL(r, C), and 
then we can take a to be a section of the associated fiber bundle with fiber Y. 
Banfield [Ba] and Mundet [MR] have given Hitchin-Kobayashi correspondences for 
these objects. 

In a future paper we will consider principal G-bundles for more general groups. 

Notation. If / : Y — > Y' is a morphism, we denote / = idx x/- If E$ is a coherent 
sheaf on X x S, we denote Es(m) := E$ ® p* x Ox{fn)- To simplify the notation, 
we will denote the complex groups GL(r, C), 0(r, C), Sp(r, C),... by GL(r), O(r), 
Sp(r). If X, Y, Z are schemes, then ir x , vr yxZ , etc... denote the corresponding 
projections from X x Y x Z. 

If P\ and P2 are two polynomials, we write Pi ~< P2 if P\(rn) < P2(m) for 
m 3> 0, and analogously for and "<". We use the convention that whenever 
"(semi) stable" and "(<)" appear in a sentence, two statements should be read: one 
with "semistable" and "<" and another with "stable" and "<". 
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Narasimhan, S. Ramanan, C.S. Seshadri and Ch. Sorger for discussions on this 
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1. Stability of tensors 

Let X be a smooth projective variety over C. Fix an ample line bundle 0x(l) ° n 
X. Fix a polynomial P of degree n = dim(X), and integers s, b, c. We will denote 
by r and d the rank and degree of a sheaf with Hilbert polynomial P. Fix a family 
{Du}u&R of locally free sheaves X parametrized by a scheme R, i.e. we fix a locally 
free sheaf V on X x R, and given a point u G R, we denote by D u the restriction to 
the slice X x u. 

Definition 1.1 (Tensor). A tensor is a triple (E,ip,u), where E is a coherent sheaf 
on X with Hilbert polynomial Pe = P, u is a point in R, and if is a homomorphism 

tp : (£®s)©c — > (det E)® b ® D Ul 

that is not identically zero. Let (E, if, u) and (F, if), v) be two tensors with Pe = Pf, 
det-E = det.F, and u = v. A homomorphism between (E,ip,u) and (F,ip,v) is 
a pair (/, a) where f : E — ► F is a homomorphism of sheaves, a G C, and the 
following diagram commutes 

(1.1) (£®s)©c 
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(det E)® b ® D u — — ^ (det F)® 6 ® D v 

where f : det E — > det F is the homomorphism induced by f . In particular, (E, (p, u) 
and (E, Xip, u) are isomorphic for A <G C*. 

Remark 1.2. We could have defined a more restrictive notion of isomorphism, 
considering only isomorphisms for which a = 1. If we do this, we obtain a different 
category: for instance, if E is simple, the set of automorphisms of (E,ip,u) is C*, 
but if we require a = 1, then the set of automorphisms is Z/(rb — s)Z (assuming 
rb — s ^ 0). If r6 — s ^ 0, even if the categories are not equivalent, the set of 
isomorphism classes will be the same (because a can be absorbed in / by changing 
/ into a l / i < rb ~ s ' > /), and then the moduli spaces will be the same. But if rb — s = 0, 
then a cannot be absorbed in /, and the set of isomorphism classes is not the same. 

Let 5 be a polynomial with deg(<5) < n = dim(X) 

(1.2) 5 = Sif 1 - 1 + 5 2 t n ' 2 + ■ ■ ■ + S n EQ[t], 

and 5(m) > for m > 0. We denote r = 5±(n — 1)!. We will define a notion 
of stability for these tensors, depending on the polarization and 5, and we will 
construct, using geometric invariant theory (GIT), a moduli space for semistable 
tensors. 

A weighted filtration (E m ,m») of a sheaf E is a filtration of sheaves 

(1.3) C E 1 C E 2 C • • • C E t C F t+1 = E, 

and positive numbers mi, rri2, ■ ■ ■ , m t > 0. Let T{ = rk(E'j). If t = 1 (one step 
filtration), then we will take mi = 1. We will denote E % = E/Ei and r l = rk(F l ). 
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The filtration is called saturated if all sheaves Ei are saturated in E, i.e. if E % is 
torsion free. 

Consider the vector of C r defined as 7 = Yll=i "Vy^j where 

k r—k 



(1.4) 7 (fc) = (k-r, ...,k-r,k,...,k) (1 < k < r - 1). 

Now let X = {1, . . . , t + 1} XS be the set of all multi-indexes / = (ii, . . . , i s ). Let jj 
be the j-th component of the vector 7, and define 

(1.5) fi((p,E.,m.) =mm{ 7rii + • • • + 7^ : ¥\( Eil ®-®E is )®c 7^0} 

Definition 1.3 (Stability). Lei <5 6e a polynomial as in (1.2). We say that (E,(p,u) 
is 5 -(semi) stable if for all weighted filtrations it is 
t 

(1.6) (^m i (rP Ei -nP))+^,£.,m.)5(^) 

i=i 

Recall that we assume that ip is not identically zero. It is easy to check that if 
(E, ip, u) is semistable, then E is torsion free. In this definition, it suffices to consider 
saturated filtrations, and with rk(^) < rk(i?j + i) for all i. 

Lemma 1.4. There is an integer A\ (depending only on P, s, b, c and V) such 
that it is enough to check the stability condition (1.6) for weighted filtrations with 
mi < Ai for all i. 

Proof. Again, let Z = {1, . . . , t+l} xs . Multi-indexes will be denoted / = (i±, . . . , i s ). 
Note that (1.5) is a piece-wise linear function of 7 € C, where C C II is the cone 
defined by 71 < ... < j r . This is because it is defined as the minimum among a 
finite set of linear functions, namely the functions j rii + • • • + 7 r - s for I G I. There 
is a decomposition of C = \J IeI Cr into a finite number of subcones 

d := {7 € C : 7r n + • ■ ■ + 7r ls < 7r t , + • • • + 7r t , for all /' € 1} 

and (1.5) is linear on each cone C/. Choose one vector 7 G Z r in each edge of each 
cone C/ . Multiply all these vectors by r, so that all their coordinates are divisible by 
r, and call this set of vectors S. All vectors in S come from a collection of weights 
rrii > 0, i = 1, . . . , t + 1, given by the formula 7 = ^=1 mn^- Hence to obtain 
the finite set S of vectors it is enough to consider a finite set of values for rrii, and 
hence there is a maximum value A\. 

Finally, it is easy to see that it is enough to check (1.6) for the weights associated 
to the vectors in S. Indeed, first note that since the first term in (1.6) is linear on 
C, then it is also linear on each C/. Then the left hand side of (1.6) is linear on each 
Ci , and hence it is enough to check it on all the edges of all the cones Ci . 

□ 

Definition 1.5 (Slope stability). We say that (E,(p,u) is slope-r- (semi) stable if E 
is torsion free, and for all weighted filtrations we have 
t 

(1.7) ( mi (r deg .Ei -rideg-E)) + fi(ip, E,, m.) r (<) 

i=i 

Recall, r = 5\(n — 1)!. As usual, we have the following implications 

slope-r-stable 5-stable =^ 
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==> (5-semistable slope- r-semistable 

The reason why we have to consider nitrations instead of just subsheaves is that 
(1.5) is not linear as a function of {mi}. But we have the following result that will 
be used in the proof of theorem 2.5. 

Lemma 1.6. Let (E,,m 9 ) be a filtration as above, and let T be a subset of T = 
{1, . . . ,t}. Let (E' 9 ,m' m ) be the subfiltration obtained by considering only those terms 
Ei for which i € T' . Then 

n(<p, E,,m,) < fi((p,E'„m',) + ^ TOjsrj. 

ieT-T' 

Proof. Let 1 = {1, . . . , t + 1} XS be the set of all multi-indexes I = . . . , i s ). Given 
a multi-index / € 1, we have 

t 

7r 4l + ■ • • + ln 3 = ^mi{sri- Ui(L) r) , 

i=l 

where Vi(L) is the number of elements k of the multi-index / = (ii,...,i s ) such 
that rfc < rj. If I is the multi-index giving minimum in (1.5), we will denote 
ei(ip, E,,m,) := Vi(L) (or just €i(E,) if the rest of the data is clear from the context). 
Then 

t 

(1.8) fi(ip,E,,m.) = y^mi(srj - €i(E,)r) . 

i=i 

We index the filtration (E'„ to'.) with T. Let I' = . . . , i' s ) £ {T U {t + 1}} Xs be 
the multi-index giving minimum for the filtration (E',,m' t ). In particular, we have 

<p\(e,®-®e.,)®c + 0. Then 

n(<p,E.,m.) = mm{7 r . +- + 7r is : ¥'|(£! il ®...®B i ,)e«= ^ 0} 

< 7ry H Vlr, 

t 

= ^2rni(sri- n{L')r) 
i=i 
t 

= ^TOi(srj - ei(E',)r) 
i=i 

= ^ mj(srj - ej(E',)r) + ^ m^sn - ei(E' m )r) 
ieT' ieT-T' 

< n((p,E'„m',) + ^2 m i sr i- 

ieT-T' 

□ 

A family of (5-(semi)stable tensors parametrized by a scheme T is a tuple (Et, (Pt, ut, 
consisting of a torsion free sheaf Et on X x T, flat over T, that restricts to a torsion 
free sheaf with Hilbert polynomial P on every slice X x t, a morphism : T — > 12, 
a line bundle iV on T and a homomorphism t^y 

(1.9) f T : (£ T ® S )® C — » (det E T f b ® u^Z? ® 
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(recall ut = idx xut) such that if we consider the restriction of this homomorphism 
on every slice X x t 

<p t :(E®°f C — (detE t f b ®D u(t) , 

the triple (E t ,<p t ,u(t)) is a (5-(semi)stable tensor for every t (in particular, ip t is not 
identically zero). 

Two families {Et, <Pt,ut, N) and (E' T N') parametrized by T are isomor- 

phic if ut = u' T and there are isomorphisms / : Et — ► E T , a : N — > TV' such that 
the induced diagram 

/ f (g)s\©c 

(£ T ® S )® C — »- (E' T ® S )® C 

<PT <p' T 

(det E T )® b ® u?*£> <8> Tr^iV — > (det ^)® 6 ®u' T V ® tt t N' 

commutes. 

Let Al^ (respectively A^f) be the contravariant functor from the category of 
complex schemes, locally of finite type, (Sch/C) to the category of sets (Sets) which 
sends a scheme T to the set of isomorphism classes of families of (5-semistable (re- 
spectively stable) tensors parametrized by T, and sends a morphism T' — > T to the 
map defined by pullback (as usual). 

Definition 1.7. A scheme Y corepresents a functor F : (Sch/C) — > (Sets) «/ 

(1) There exists a natural transformation f : F —*Y_ (where Y_ is the functor of 
points represented by Y). 

(2) For every scheme Y' and natural transformation f : F —> Y^_, there exists a 
unique g :Y_^>Y[_ such that f factors through f . 

IfY exists, then it is unique up to unique isomorphism. If furthermore /(SpecC) : 
F(SpecC) — > Y is bijective, we say that Y is a coarse moduli space. 

We will construct schemes QJt^, SUtf corepresenting the functors M s and M\. 
In general Wis will not be a coarse moduli space, because nonisomorphic tensors 
could correspond to the same point in Wis. As usual, we declare two such tensors 
S-equivalent, and then Wis becomes a coarse moduli space for the functor of S- 
equivalence classes of tensors. 

Theorem 1.8. FixP, s, b, c and a family V of locally free sheaves onX parametrized 
by a scheme R. Let d be the degree of a coherent sheaf whose Hilbert polynomial is 
P. Let 5 be a polynomial as in (1.2). 

There exists a coarse moduli space Wis, projective overPic d (X)xR, of S- equivalence 
classes of 5-semistable tensors. The closed points of Wis correspond to S-equivalence 
classes of 5-semistable tensors. There is an open set Wl s 5 corresponding to 5 -stable 
tensors. Points in this open set correspond to isomorphism classes of 5-stable ten- 
sors. 

In proposition 4.1 we give a criterion to decide when two tensors are S-equivalent. 
Theorem 1.8 will be proved in section 4. 

Remark 1.9. Note that to define the functors we have used isomorphism classes of 
families, but usually one uses equivalence classes, declaring two families equivalent 
if they differ by the pullback of a line bundle M on T. As a result, in general the 
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functors that we have defined will not be sheaves. The sheafified functors will be 
the same (this follows from the fact that if we shrink T then M will be trivial), and 
hence the corresponding moduli spaces are the same, because a scheme corepresents 
a functor if and only if it corepresents its sheafification (see [Si, p. 60]). 

2. BOUNDEDNESS 

The objective of this section is theorem 2.5, where we reformulate the stability 
condition for tensors using some boundedness results. We start with some well 
known results. See [Si, cor 1.7] (also [H-L2, lemma 2.2]), [Gr, lemma 2.5] and [Ma]. 

Lemma 2.1 (Simpson). Let r > be an integer. Then there exist a constant B 
with the following property: for every torsion free sheaf E with < rk(E) < r, we 
have 

h°(E) < -^((rk(^)-l)([^ max (ii;) + J B] + ) n +([^ min ( J E)+i?] + )"), 

where g = degOxiX), [x]+ = max{0, x}, and fJ,max{E) (respectively fi m i n (E)) is the 
maximum (respectively minimum) slope of the Mumford-semistable factors of the 
Harder-Narasimhan filtration of E. 

Lemma 2.2 (Grothendieck). Let T be a bounded set of sheaves E. The set of 
torsion free quotients E" of the sheaves E inT with | deg(-E")| < C" for some fixed 
constant C" , is bounded. 

Theorem 2.3 (Maruyama). The family of sheaves E with fixed Hilbert polynomial 
P and such that ^ max (-E) < C for a fixed constant C , is bounded. 

Corollary 2.4. The set of 5-semistable tensors (E,ip,u) with fixed Hilbert polyno- 
mial is bounded. 

Proof. Follows from theorem 2.3 and an easy calculation. □ 

The main theorem of this section is 

Theorem 2.5. There is an integer No such that if m > Nq, the following properties 
of tensors {E, ip, u) with E torsion free and Pe = P, are equivalent. 

(1) (E,ip,u) is (semi)stable. 

(2) P{m) < h°(E(m)) and for every weighted filtration (E,,m,) as in (1.3) 

t 

(j2 m i( rh °( E i( m )) ~ r i p ( m ))) +v(<P,E.,m.)5(m) (<) 

i=i 

(3) For every weighted filtration (E,,m,) as in (1.3) 

t 

(J2 m i{ rip ( m ) -rhP{E\m)))^ + n(ip, E„m.) 5{m) (<) 
i=i 

Furthermore, for any tensor (E, if, u) satisfying these conditions, E is m-regular. 

Recall that a sheaf E is called m-regular if h l (E(m — i)) = for i > 0. If E is 
m-regular, then E(m) is generated by global sections, and it is m'-regular for any 
m! > m. The set of tensors (E,<p,u), with E torsion free and Pe = P, satisfying 
the weak version of conditions 1-3 will be called 5 s , S' m , and S!^. 
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Lemma 2.6. There are integers N±, C such that if (E,(p,u) belongs to S = S s U 
\J m>Nl S'm, then for all saturated weighted filtrations the following holds for all i: 

(2.1) deg(^) - nn a < C, 

(where fi s = (d — sr)/r) and either —C < deg(-Ej) — rj/i s or 

(1) rh°(Ei(m)) < ri{P{m) - s8(m)), if (E,<p,u) G S s and m>Ni. 

(2) r\P -s5)^ r(P Ei -s5),if (E, <p, u) G U m > Wl 

Proof. Let B be as in lemma 2.1. Choose C large enough so that C > st and the 
leading coefficient of G — (P — s5)/r is negative, where 

( 2 - 2 ) G(m) = ((1 - \) {n s + sr + mg + B) n + \ ( M- - ±C + mg + 

Choose A^i large enough so that for m > Aq, 

(2.3) 5{m) > 0, 

C 

(2.4) /i s +mg + B > 0, 

r 

(2.5) G(m) - (P(m) - s5(m))/r < 0. 

Since the filtration is assumed to be saturated, and since E is torsion free, we have 
< Ti < r. 

Case 1. Suppose (E,(p,u) G S s . For each i, consider the one step filtration E^ C 
E. The leading coefficient of the semistability condition applied to this filtration, 
together with C > st, implies (2.1). 

Let Ei. ma x C Ei be the term in the Harder-Narasimhan filtration with maximal 
slope. Then the same argument applied to £j iinax gives 

(2.6) /Umax(£j) = M-E^max) < fJ>s + ST. 

Now assume that the first alternative does not hold, i.e. 

-C > deg(Ei) - rm s . 

This gives 

C 

(2.7) Hrmn(Ei) < ^{Ei) < fX s - - . 

Combining lemma 2.1 with (2.4), (2.6), (2.7) and (2.5), we have 

rh°(Ei(m)) < n(P(m) - s 6(m)). 

Case 2. Suppose (E, ip, u) G S" t for some m > N\. For each i, consider the quotient 
E % = E/E{. Let E* nin be the last factor of the Harder-Narasimhan filtration of E l 

(i.e. M^mm) = Mmin(^))- Let E ' be the kernel 

E' E £ min 0, 

and consider the one step filtration E' C E. Equations (2.2) and (2.4) imply that 
< G{m). Then a short calculation using (2.5), the fact that (E,ip,u) G S'^, (2.3) 
and lemma 2.1 shows 



G(m) < * ggff } < -^(^m + m9 + Br. 
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It can be seen that if this inequality of polynomials holds for some m > N±, then it 
holds for all larger values of m, hence choosing m large enough and looking at the 
coefficients, we have 

1 C 

Mmin(^) > lis+ (l--)sr-^. 

A short calculation using this, fi, ia hi{E % ) < ^(E 2 ) and < rk(E % ) < rk(E) (hence 
rk(E) > 1), yields (2.1). 

Now assume that the first alternative does not hold, i.e. 

-C > deg(Ei) - nii s . 

It follows that r % pu s < deg(E l ) — st, and hence 

r^P-sS) < r{P El - sS). 

□ 

Lemma 2.7. The set S = S s U Um>Ai ^ s bounded. 

Proof. Let (E, ip, u) G S. Let E' be a subsheaf of E, and E the saturated subsheaf 
of E generated by E' . Using lemma 2.6 

ME') r k(E) rk(^) 

Then by Maruyama's theorem 2.3, the set S is bounded. □ 

Lemma 2.8. Let Sq be the set of sheaves E' such that E' is a saturated subsheaf of 
E for some (E, ip, u) £ S, and furthermore 

(deg^O-^l < C. 

Then So is bounded. 

Proof. Let E' G <So- The sheaf E" = E/E' is torsion free, and |deg(i? // )| is bounded 
because the set S is bounded and 

|deg(£")| < |deg(£7)| + |deg(^)| < max | deg(S)| + C + r\fi s \. 

Then by Grothendieck's lemma 2.2, the set of sheaves E" obtained in this way is 
bounded, and hence also Sq. □ 

Lemma 2.9. There is an integer N \ such that for every weighted filtration (E,,m,) 
as in (1.3) with E; L £ <So, the inequality of polynomials 

t 

(Y,M rP E> ~nP)) +n(<p,E.,m.) 5 (r<) 
1=1 

holds if and only if it holds for a particular value of m > N2. 

Proof. Since So is bounded, the set that consists of the polynomials 5, Po, r'Po and 
Pei for E' G Sq is finite. On the other hand, lemma 1.4 implies that we only need 
to consider a finite number of values for rrij, hence the result follows. □ 
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Proof of theorem 2.5. Let No > maxjiVi, N2} and such that all sheaves in S and So 
are iVo-regular, and Pi <g> • • • % E s is sA^-regular for all Pi, . . . , E s in Sq. 

2. =>■ 3. Let (E,ip,u) G S' m . Consider a weighted filtration (P., to.). Then 

t 

( ^ m ! (r i P(m) - rP°(P*(m)))) + ^(93, P., m.) <5(m) < 
i=i 

t 

(^(^(^HJ-riPM)) +A*(^£?.,m.)(y(m) (<) 

i=i 

1. => 2. Let (E,(p,u) G 5 s and consider a saturated weighted filtration as in (1.3). 
Since £ is iV -regular, P(m) = h°(E(m)). If P* G <S , then P Bi (m) = h°(Ei(m)). If 
Pj ^ So, then the second alternative of lemma 2.6 holds, and then 

rh°(Ei(m)) < rj(P(m) - s 8{m)) . 

Let T' C T = {1, . . . , i} be the subset of those i for which Pj G <So- Let (P£, to'.) be 
the corresponding subfiltration. Lemma 1.6 and a short calculation shows that 
t 

(2.8) (^mi{rh°{Ei{m)) -rjP(m))) + n{<P, E.,m.) 5{m) < 

i=l 

( ^ m i( r p Ei(m) -nP(m))j + n{(p,E'„m',)5{m) + 

ieT' 

( ^2 m A r ^°(Pi( m )) ~~ rjP(m) — srj5(m))^ < 
ier-T' 

(^ m! ( r P B ,(ra)-r l P(m)))+ / i(^E:,m:)iH (<) 

ieT' 

The condition that Pj is saturated can be dropped, since h , (Pj(m)) < /i°(Pj(m)) 
and fj,(<p, E,,m») = (i(ip, P # , to.), where Pj is the saturated subsheaf generated by 
Pj in P. 

3. =>■ 1. Let (E,ip,u) G 5^ and consider a saturated weighted filtration (P., to.). 
Since P is N -regular, P(m) = h°(E(m)). If P G 5b, then P s .(m) = ft°(Pj(m)). 
Hence hypothesis 3 applied to the subfiltration (E' 9 ,m' 9 ) obtained by those terms 
such that Pj G So implies 

( m{r l P(m)-rP & {m)))+^,E',,m',)5(m) (<)0. 

This is equivalent to 

( mi(rP Ei (m) - r^P(m))) + //(<^, P^, m'.) <5(m) (<) 0, 

and by lemma 2.9, this is in turn equivalent to 

(2.9) ( Y, m i (rP Ei -r i P)) +fi{<p,K, O H±) ® ■ 

If E{ £ So, then the second alternative of lemma 2.6 holds, and then 

(2.10) rP Ei -riP + sr { b < 0. 
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Using lemma 1.6, (2.9) and (2.10) 

^mi(rP Ei - ri P)) +n(<p,E.,m.)6(l)0. 

i=i 

Again, we can drop the condition that the filtration is saturated, and this finishes 
the proof of theorem 2.5 

□ 

Corollary 2.10. Let (E,<p,u) be 5-semistable, m > No, and assume that there is a 
weighted filtration (P.,m.) with 
t 

(^m;(r/i°(P(m)) - r;P(m))) + n(<p,E.,m.)6(m) = 0. 

i=\ 

Then E^ G So and h°(Ei(m)) = Pe^m) /or a// i. 

Proof. By the proof of the part (1 =4* 2) of theorem 2.5, if we have this equality then 
all inequalities in (2.8) are equalities, hence T = T, E; L G 5o for all i, and the result 
follows. □ 

Note that in theorem 2.5 we are assuming that E is torsion free. To handle the 
general case, we will use the following lemma 

Lemma 2.11. Fix u G R. Let (E,ip,u) be a tensor. Assume that there is a family 
(E t , (ft,u)t£C parametrized by a smooth curve C such that (P , tp , u) = (E, (p, u) and 
E t is torsion free for t ^ 0. Then there exists a tensor (F,ip,u), a homomorphism 

(E,<p,u) — ► (F,ip,u) 

such that F is torsion free with Pe = Pf, o-nd an exact sequence 

— > T(E) — ► E -^-> F, 
where T(E) is the torsion subsheaf of E. 

Proof. The family is given by a tuple {Eq , <pc , , N) as in (1.9), where uq is the 
constant map from C to R with constant value u. Shrinking C, we can assume 
that N is trivial. Let U = (X x C) - Supp(T(£ ))- Let Pc = 3*{E c \u)- Since 
it has no C-torsion, Fq is flat over C. The natural map (3 : Eq — > Pc* is an 
isomorphism on U, hence we have a homomorphism tpu := (fc\u on U, and this 
extends to a homomorphism ipc on X x C because uc*V is locally free. Finally 
define (P, ip) = (Po,?/>o), and let (3 be the homomorphism induced by (3. □ 

3. GIT CONSTRUCTION 

Let N > No be large enough so that for allz > 0, all line bundles L of degree 
d, all locally free sheaves D u in the family parametrized by R, and all m > N, we 
have h l (L® b <g> D u {sm)) = and L® b <g) D u (sm) is generated by global sections. 

Fix m > N and let V be a vector space of dimension p = P(m). The choice 
of m implies that if (P, <p, u) is (5-semistable, then E(m) is generated by global 
sections and h l {E{m)) = for i > 0. Let (g,E, </?, u) be a tuple where (E,(p,u) is 
a (5-semistable tensor and g is an isomorphism g : V — > H°(E(m)). This induces a 
quotient 

(3.1) g : V(g>0 x (-m) ^P- 
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Let H be the Hilbert scheme of quotients of V <8> Ox(—m) with Hilbert polynomial 
P. Let / > m be an integer, and W = H°(Ox{l — m)). The quotient q induces 
homomorphisms 

V®O x (l-m) -» £(Z) 

F<g>VF -» H°(E(l)) 
f\ p ^(V0W) -» f\ p ^H°(E(l)) C 

If Z is large enough, these homomorphisms are surjective, and give Grothendieck's 
embedding 

W — ► P(/\ p{l) (V v ®W V )), 

and hence a very ample line bundle 0-^(1) on H (depending on m and I). 
The tuple (5, P, <p, u) induces a linear map 

(3.2) <$>:{V® S )® C _ » tf°((£(m) 0s ) ec ) — ► i?°((det i?)® 6 (8) D u (sm)). 

Fix a Poincare bundle P on J x I, where J = Pic rf (X). Fix an isomorphism 

: det(E) — P| {de t(£)}xx- 

Then $ induces a quotient 

® #Vlf d V)}xx ® A.(«n)) v — C. 
Choosing a different isomorphism /? will only change this quotient by a scalar, so 
we get a well defined point [$>] in P, where P is the projective bundle over J x R 
defined as 

P = P(((^)®T ® vr Jxi? ,(^ xJ P^ ® 7r XxR D(sm))) ^JxR. 

where vr XxJ (respectively ir JxR ,...) denotes the natural projection from X x J x R 
to X x J (respectively J x R,...). Recall that P(m) := V ® -K x Ox{m). Note 
that 7T JxR ^ {tt X x jP® b ® 7r^ xR P(sm)) is locally free because of the choice of m. 
Replacing P with another Poincare bundle defined by tensoring with the pullback 
of a sufficiently positive line bundle on J, we can assume that Op(\) is very ample 
(this line bundle depends on m). 

A point (q, [$]) € H x P associated to a tuple (g, E,(p,u) has the property that 
the homomorphism <I> in (3.2) composed with evaluation factors as 



(3.3) 



(y® s )® c ® o x 



(P(m) 0s )® c 



#°((det £)® b ® D u (sm)) ® O x 



(det£)® b ® fl„(sm) 

Consider the relative version of the homomorphisms in (3.3), i.e. the commutative 
diagram on X x TC x P 



(3.4) 




XxHxP ~ 

A := Pxxj'P® h ®PxxiP®PxOx{srn) 
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where p Xx j (respectively p x , ...) denotes the natural projection from X x 7i x P 
to X x J (respectively X, ...), En is the tautological sheaf on X x H, and &HxP is 
the relative version of the composition ev o in diagram (3.3). 



The points (q, [<£]) where the restriction &Hxp\xx(q,[$]) factors through (E{ 



m) 



(as in 3.3) are the points where f\xx(q,[$}) is identically zero. We will need the fol- 
lowing 

Lemma 3.1. Let Y be a scheme, and let f : Q — > T be a homomorphism of coherent 
sheaves on X x Y . Assume that T is flat over Y . Then there is a unique closed 
subscheme Z satisfying the following universal property: given a Cartesian diagram 

X x S^^X x Y 

PS 



S ^Y 

h* f = if and only if h factors through Z. 

Proof. Uniqueness is clear. To show existence, assume that O x (l) is very ample 
(taking a multiple if necessary) and let p : X x Y — > Y be the projection to the 
second factor. Since T is Y-flat, taking m! large enough, p 1f P{m') is locally free 
(recall ^{m') = T <8> P* x O x (m')). The question is local on Y, so we can assume, 
shrinking Y if necessary, that Y = Spec A and p^!F(m') is given by a free yl-module. 
Now, since Y is afHne, the homomorphism 

Pj{rri) : p*G{m') — > p*F{m') 

of sheaves on Y is equivalent to a homomorphism of A-modules 

M {h r4 n) A © • • • A 

The zero locus of fa is defined by the ideal h C A image of , thus the zero scheme 
of (/i, . . . , f n ) is given by the ideal / = Yl hence Z' m , is a closed subscheme. 

Since is very ample, if m" > m' we have an injection p^T{m') > p^Tim!'} 

(and analogously for Q), hence Z m » C Z m i, and since Y is noetherian, there exists 
N' such that, if m! > N', we get a scheme Z independent of m'. 

To check the universal property first we will show that if h f = then h factors 
through Z. Since the question is local on S, we can take S = Spec(S), Y = Spec(^4), 
and the morphism h is locally given by a ring homomorphism A — > B. Since T is 
flat over Y, for m' large enough the natural homomorphism a : h*p*P(m') — ► 
ps*h J-(m') (defined as in [Ha, Th. Ill 9.3.1]) is an isomorphism. Indeed, for m' 
sufficiently large, IP{X, F y {m')) = and H l {xX \F{m')) s ) = for all points y G 
Y, s £ S and i > 0, and since T is flat, this implies that h*p*!F(m') and ps*h* jF(m') 
are locally free. Then, to prove that the homomorphism a is an isomorphism, it is 
enough to check it at the fiber of every s G S, but this follows from [Ha, Th. Ill 
12.11] or [Mu, II §5 Cor. 3]. 

Hence the commutativity of the diagram 

* ps „h* f(m')=0 # 

PS*h Q(m') ^Ps*h Tim') 



nl l\ h *P*f( m ') ,* nr , , 
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implies that h*p*f(m') = 0. This means that for all i, in the diagram 

k 



M 



A 



A/h 







1 k®B V 

M® A B B 



A/h ® A B ^ 



it is fi <g> B = 0. Hence the image Ij of fi is in the kernel J of A —> B. Therefore 
I C J, hence A — > P factors through ^4 — > A/7, which means that /i : S" — > y factors 
through Z. 

Now we show that if we take S = Z and /i : Z y the inclusion, then h f = 0. 
By definition of Z we have h*p*f(m') = for any to' with m/ > N'. Showing that 
h f = is equivalent to showing that 

h*f(m') : h*g(m') — » 7T.F(m') 

is zero for some to'. Take to' large enough so that ev : p*p*G{m') — > Q{m!) is 
surjective. By the right exactness of /i the homomorphism /i ev is still surjective. 
The commutative diagram 



h G{m>) 

77* ' 
h ev 

h p*p*Q{m!) 



h /(m') 



h*p*p*f{m') 



> /i .F(m') 

A 



h* p*p 1r T{'m') 



Pgh*p*f(m')=0 

p* s h*p*Q(m ) > p^lfp^F^m ) 



implies h* f(m!) = 0, hence /i* f = 0. 



□ 



Let Z' be the scheme given by this lemma for Y = TLx P and the homomorphism 
f : IC —> A. Then i f = 0, and there is a commutative diagram on I x 2' 



(3.5) 




( j *P*xxn E n( m ) 







and hence there is a universal family of based tensors parametrized by Z' 

(3.6) <p z , : E z ,® s — ► (det E z ,)® b ® p|,7V 

Given a point (g, [$>]) in Z' ', using the tautological family (3.6) we can recover 
the tuple (q, E, <p,u) up to isomorphism, and if H°(q(m)) : V — > H°(E(m)) is 
an isomorphism, then we recover the tuple (5 = H°(q(m)),E,<p,u) up to isomor- 
phism, i.e. if (g',E',<f',u') is another tuple corresponding to the same point, then 
there exists an isomorphism (/, a) between (E,tp,u) and (E',<p',u') as in (1.1), and 
H°(f(m))oq = q>. 

Let Z C Z' be the closure of the points associated to 5-semistable tensors. Let 
p n and p p be the projections of Z to H and P, and define a polarization on Z by 

(3.7) Oz(ni,n 2 ) := p* n O n ( ni ) ® ppO P {n 2 ), 
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where n\ and n<i are integers with 

U2 _ P{l)5(m) - 5(l)P(m) 
ri\ P(ra) — s5(m) 

The projective scheme Z is preserved by the natural SL(V) action, and this action 
has a natural linearization on Oz{n\, 71-2), using the natural linearizations on 0^(1) 
and O p (1). 

We have seen that the points of Z for which H°(q(m)) is an isomorphism corre- 
spond (up to isomorphism) to tuples (g, E, <p, u), where g is an isomorphism between 
V and H°(E(m)). To get rid of the choice of g, we have to take the quotient by 
GL(V), but if A £ C*, (g,E,ip,u) and (\g, E, <p,u) correspond to the same point, 
and hence it is enough to divide by the action of SL(V). In fact, the moduli space 
will be the GIT quotient of Z by SL(V). 

In proposition 3.4, we will identify the GIT- (semi) stable points in Z using the 
Hilbert-Mumford criterion. In theorem 3.6 we relate nitrations of sheaves with 
nitrations of the vector space V to prove that GIT-(semi)stable points of Z coincide 
with the points associated to J-(semi)stable points. 

A nonconstant group homomorphism A : C* — ► SL(V) is called a one-parameter 
subgroup of SL(V). If SL(V) acts on a projective scheme Y with a given lineariza- 
tion, we denote by fi(y, A) the minimum weight of the action of A on y G Y. 

A weighted filtration (V,,m,) of the vector space V is a filtration of vector spaces 

(3.9) 0CV 1 CV 2 C ■■■ CV t CV t+1 =V, 

and positive numbers mi, mi, ■ ■ ■ , m t > 0. If t = 1 (one step filtration), then we 
will take mi = 1. Consider the vector of C7 defined as T = ^- =1 mir( dim ^)), where 

k p—k 

r( fc ) = (k- P ,."..,k-p,k~^~k) (i<k< P ). 

Now let 1 = {1, . . . , t + 1} XS be the set of all multi-indexes / = (ii, . . . , i 8 ), and 
define 

(3.10) ii(§,V;m.) = min{r dimyii + ••• +r dimVis : $|(y il ®... (8 y ij )©c + } 

As we did in the proof of lemma 1.6, if / is the multi-index giving minimum in (3.10), 
we will denote by £j($, V,, m # ) (or just £i(V») if the rest of the data is clear from the 
context) the number of elements k of the multi-index I such that dim < dimV^. 
Then we have, as in (1.8) 

t 

(3.11) ^($,V r .,m.) = ^m;(sdimVi - e^V.) dimV) . 

i=i 

Given a subspace V C V and a quotient q : V <S> Oi(-m) E, we define the 
subsheaf Ey> of E as the image of the restriction of q to V 

V®O x {-m) ^ E 

f 

V ®O x {-m) ^E v , 

In particular, Eyi(m) is generated by global sections. 



(3.8) 
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On the other hand, if the quotient q : V ® Ox(—m) -» E induces an injection 
V <^-> H°(E(m)) (we will later show that all quotients coming from GIT-semistable 
points of Z satisfy this property), and if E' C E is a subsheaf, we define 

V E > = V nH°(E'(m)). 

The following two lemmas are easy to check 

Lemma 3.2. Given a point (q, [$>]) G Z such that q induces an injection V 
H°(E(m)), and a weighted filtration (E m ,m,) of E, we have: 

(1) E VEi C Ei 

(2) If <p\( Eil ®-®E is )®c = 0, then $1(1^®-®^,)®" = 

( 3 ) Ya=i -miei(p,E,,m,) < Ya=i - m i e i(^> V E., m,) 

Furthermore, if q induces an isomorphism V = H°(E(m)), all Ei are m-regular and 
all Ei x <g) • • • ® Ei s are sm-regular, then (1) becomes an equality, (2) becomes "if and 
only if" and (3) an equality. 

Lemma 3.3. Given a point (q, [$>]) € Z such that q induces an injection V 
H°(E(m)), and a weighted filtration (V,,m.) ofV, we have: 

(1) Vi C V Evi 

(2) ¥\{E Vii ®-®E Vt3 )®- = if and only if <f>\( Vil ®-®V is )®c = 

( 3 ) Ya=i -mi€i{ip,E v .,m,) = £- =1 -m, V., m.) 

Proposition 3.4. For sufficiently large I, the point (q, [<£]) in Z is GIT- (semi) stable 
with respect to Oz(»i,"2) if and only if for every weighted filtration (V,,m,) ofV 

t 

(3.12) ni (j2 m i{ dimV i P ( l ) ~ dimV P E Vi (l))) + n 2 ^,V.,m.) (<) 0. 

i=l 

Furthermore, there is an integer A 2 (depending only on m, P, s, b, c and V) such 
that it is enough to consider weighted filtrations with rrii < A 2 . 

Proof. Given to, the sheaves Ey for V' C V form a bounded family, so if I is large 
enough, we will have 

dimq'(V®W) = h°(E v ,(l)) = P Ev ,{l) 

for all subspaces V' C V. By the Hilbert-Mumford criterion, a point is GIT- 
(semi)stable if and only if for all one-parameter subgroups A of SL(V), 

H((q, [$]), A) = n lf i(q, A) + n 2 /*([$], A) (<) 0. 

A one-parameter subgroup of SL(V) is equivalent to a basis {e\, . . . , e p } of V and a 
vector T = (ri, . . . , F p ) € C p with T± < . . . < T p . This defines a weighted filtration 
(V., to,) of V as follows: let Ai < . . . < Xt+i be the different values of T^, let Vi be 
the vector space generated by all such that < Aj, and let m; = (Aj+i — \i)/p. 
Denote T = {1, . . . ,t + l} xP «. We have ([Si] or [H-L2]) 

H(q,\) = mm{r dimyii H h T dimV . p ^ : q"\( Vil ®W) A- A(V lp(l) ®W) + °} 

i 

= J] mi ( dim V|P(Z) - dimV P Ev .(l)) 

i=l 



TENSORS AND ORTHOGONAL SHEAVES 17 

and 

A) = mm {r dim y H + • • • + r dim y is : $|(y il0 ... y is) ec + 0} 

The last statement follows from an argument similar to the proof of lemma 1.4, 
with 27 replaced by TP. 

□ 

Proposition 3.5. The point (q, [<&]) is GIT- (semi) stable if and only if for all 
weighted filtrations (E,,m,) of E 



T,\=im[{^V Ei -e l {E.)5{m)){P-s5) 

- (P Ei - ei(E.)6) {dimV - sS(m)j) (r<) 



Furthermore, if(q, [<&]) is GIT-semistable, then the induced map f q :V—> H°(E(m)) 
is infective. 

Proof. First we prove that if (q, [<£]) is GIT-semistable, then the induced linear map 
f q is injective. Let V' be its kernel and consider the filtration V' C V. We have 
Eyi = and ^(<I>, V' C V) = sdim V'. Applying proposition 3.4 we have 

m dim V'P(l) + n 2 s dim V' < 0, 

and hence V = 0. 

Using (3.8) and (3.11), the inequality of proposition 3.4 becomes 



(3.14) 



E -=i rm ( ( dim Vi - ei(V.) <5(m)) (P(Z) - s<5(Z)) 

- (P Bv . (Z) - (dimV - s<5(m))) (<) 



An argument similar to lemma 2.9 (using instead of Ai) shows that we can take 
I large enough (depending only on m, s, b, c, P, T> and 5), so that this inequality 
holds for I if and only if it holds as an inequality of polynomials. 

Now assume that (q, [$>]) is GIT-(semi)stable. Take a weighted filtration (E,,m,) 
of E. Then lemma 3.2 and (3.14) applied to the associated weighted filtration 
(V E .,m.) of V give (3.13). 

On the other hand, assume that (3.13) holds. Take a weighted filtration (V,, m.) 
of V. Then lemma 3.3 and (3.13) applied to the associated weighted filtration 
(E v .,m.) of E give (3.14), and it follows that (q, [<&]) is GIT-(semi)stable. 

□ 

Theorem 3.6. Assume m > N . For I sufficiently large, a point (q, [<£]) in Z is 
GIT- (semi) stable if and only if the corresponding tensor (E,(p,u) is 5 -(semi) stable 
and the linear map f q : V — ► H°(E(m)) induced by q is an isomorphism. 

Proof. We prove this in two steps: 

Step 1. (<?, [$>]) GIT-semistable => (E,(p,u) 5-semistable and q induces an iso- 
morphism V = H°(E(m)). 

The leading coefficient of (3.13) gives 

t 

mi ({ dim V Ei — ej{E t )5{mj)r — dim V — s5(m))^) < 0. 

i=l 
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Note that even if (q, [$>]) is GIT-stable, here we only get weak inequality. This 
implies 

t 

(3.15) (^rrii^Pim) - rh° (E* (m)))^) + n(<p, E.,m.)5(m) < 0. 

i=i 

To be able to apply theorem 2.5, we still need to show that E is torsion free. By 
lemma 2.11, there exists a tensor (F,ijj,u) with F torsion free such that Pe = Pf 
and an exact sequence 

— ► T{E) — ► E -^-> F . 

Consider a weighted filtration (F.,m.) of F. Let F l = F/Fi, and let E 1 be the 
image of E in F l . Let E^ be the kernel of E —> E l . Then rk(Fj) = rk(Ei) = j-j, 
h°(F z (m)) > h®(E l (m)), and F,,m,) = n(ip, E,,m,). Using this and applying 

(3.15) to Ei we get 

t 

(J2 m i{ rip ( m ) -rh°(F\m)))^ + fi(tp, F.,m.) 5{m) < 0, 
i=l 

and hence theorem 2.5 implies that (F, ip, u) is <5-semistable. 

Next we will show that T(E) = 0, and hence, since Pe = Pf, we will conclude 
that (E, ip, u) is isomorphic to (F, ip, u). Define E" to be the image of E in F. Then 

P(m) - sS(m) = h°(F(m)) - sS(m) > h°(E"(m)) - sS(m) > P(m) - s5(m), 

where the last inequality follows from (3.15) applied to the one step filtration T(E) C 
E. Hence equality holds at all places and h°(F(m)) = h°(E"(m)). Since F is 
globally generated, F = E" , and hence T(E) = 0. 

Finally, we have seen that f q is injective, and since (E, ip) is 5-semistable, dim V = 
h°(E(m)), hence f q is an isomorphism. 

Step 2. (E,(p,u) 5-stable (respectively strictly 5-semistable) and q induces an iso- 
morphism f q :V = H°(E(m)) (q, [$]) GIT-stable (respectively strictly semistable) . 

Since f q is an isomorphism, we have Ve< = H°(E'(m)) for any subsheaf E' C E. 
Then theorem 2.5 implies that for all weighted nitrations 

t 

(3.16) (^nH(rdhnV Ei -riP(m))}+»(<p,E.,m.)6(m) (<) 

i=i 

If the inequality is strict, then 

E*=i ™i ( ( dim V Ei - e t (E.)6(m)) (P - sS) 

- (p Ei - ei (E.)5)( dimV - s5(m))^j ~< 0. 

If (E, ip, u) is strictly (5-semistable, by theorem 2.5 there is a filtration giving equality 
in (3.16), then corollary 2.10 implies that h°(Ei(m)) = Pg^m), and by lemma 2.9 

t 

(^2mi(rP Ei -r i pj)+ f i(<p,E.,m.)6 = 0, 
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and a short calculation using this and (3.16) gives 

E*=i m i ( ( dim V Ei - et (E.)S(m)) (P - sS) 

- (p Et - ei (E.)5) (dimV - s5(m))^j = 0. 
So we finish by using proposition 3.5. 

□ 

Given a one-parameter subgroup of SL(V), choose a basis {ej} of V where it has 
a diagonal form 

ai a2—a\ at+l—at 
/ > ' " > ' * \ 

diag(Ai, . . . , Ai, A2, • • . , A2, . . . , \t+i, ■ ■ ■ , Aj+i) 

This gives a weighted filtration (V,,m,) of V = H°(E(m)) (where V{ is the linear 
span of {ej} with j < ai, and mi = (Aj + i — \i)/p) and a splitting V = (BV l of 
this filtration (with V 1 the linear span of {ej} with Oj_i < j < ai). Defining 
= q(Yi ® Ojf(-m)) we obtain a weigted filtration (E 9 ,m 9 ) of E. 

Now let (E 9 ,m 9 ) be a weigted filtration of E and V = ®V % a splitting of the 
filtration V { = H°(E(m)). This gives a one-parameter subgroup A of SL(U), defined 
as v l 1— > t Xt v l for v l G V" 1 , with Aj such that rrij = (Aj+i — A,) /p and dim U l = 0. 

The following proposition will be used to prove the criterion for S-equivalence. 

Proposition 3.7. Assume m > N. Let (E,ip,u) be a 5-semistable tensor, f : 
V = H°(E(m)) an isomorphism, and let Let (q, [<&]) G Z be the corresponding GIT- 
semistable point. The above construction gives a bijection between one-parameter 
subgroups o/SL(V) with n((q, [<£]), A) = on the one hand, and weighted filtrations 
(E, , m, ) of E with 

t 

(j2 m i( rP Ei -nP)) +n(<p,E.,m.) 5 = 

i=l 

together with a splitting of the filtration H°(E m (m)) ofV = H°(E(m)) on the other 
hand. 

Proof. Let A be a one-parameter subgroup of SL(V) with fi((q, [$]), A) = 0. The 
proof of proposition 3.4 then gives equality in (3.12). Using (3.8) (relationship 
between 712/ n\ and 5), (3.11) (relationship between e(V.) and /u(U,)) and lemma 3.3 
(relationship between e(V,) and e(Ev,)), this equality becomes 

ZUm^idimVi - €i{Ev.)S(m)) (P(l) - sS(l)) 

- (P Evi (I) - e l (E v .)5{l)) (dim V - s8{m))) = 

We have chosen I so large that this holds if and only if it holds as a polynomial in 
I, hence taking the leading coefficient in I we obtain 

t 

vrij ( ( dim Vj — ej(Ey t )5(m))r — rj[dimV — s5(m))^J = 0. 

i=l 

where r\ = rk£V; and r = vkE. Using (1.8), this is 

t 

( ^! m i{ r dim Vj — rjP(m))^J + /x(y, Ey m , m,) 5(m) = 
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By lemma 3.3, Vi C Ve v . = H G \Ey i (m)) , hence 
t 

(j2 m i( rh °( E Vi(™)) ~nP(rn))^ + Li((p,E v .,m.)5(m) > 

i=l 

but by theorem 2.5 this must be nonpositive, hence Vi = H°(Evi(m)) = Ve v ., 

and the last inequality is an equality. By corollary 2.10, E± £ So, and hence 
h P(Ev i (m)) = P{Ev l {m)) for all i, and then lemma 2.9 gives 

t 

(^2mi(rP Ev .-riP)^+fj,((p,E v<t ,m.)6 = 
i=i 

Conversely, let (E 9 ,m 9 ) be a filtration with 

t 

(3.17) (j]m i (rP^-r i P))+A*(^£?.,m.)(J = 

i=i 

together with a splitting of the filtration H°(Ei(m)) oi V = H°(E(m)), and let 
A be the associated one-parameter subgroup of SL(V). Equation (3.17) gives in 
particular 

t 

(^2mi(rP Ei (m) - j-jP(m))) + (j,(<p,E m ,m.) 5(m) = 
i=i 

By the proof of implication 3. =4> 1. in theorem 2.5, since we get an equality, it is 
Ei £ 5o for all i, hence P Ei (m) = h Q {Ei{m)) = dimVE, for all i, and the previous 
equality becomes 

t 

(3.18) mi (r dim Vki ~n dim y)) +n((p,E„ m.)6(m) =0 
i=i 

Furthermore, the strong version of lemma 3.2 gives Ei = Ey E . Using (3.18) and 
(3.17), together with (1.8) and the strong form of lemma 3.2, we obtain 

t 

Y, m *(( dim Vki - £i (Vk. )<y(m)) (P - s<5) - (P Et - €i (Ve.)S) ( dim V - s6(m)) ) = 
i=i 

Hence, we also get after evaluating this polynomial in /, but by the proofs of 
propositions 3.4 and 3.5, this is equal to n((q, [$]), A). 

We have seen that Vi = Ve v . and Ei = Ey E ., and it is easy to check that this 
gives a bijection. 

□ 

4. Proof of theorem 1.8 

Proof of theorem 1.8. The main ingredient of the proof is theorem 3.6, showing that 
GIT-(semi)stable points correspond to <5-(semi)stable tensors. 

Using the notation of section 3, let 9Jls (respectively 9Jl|) be the GIT quotient of Z 
(respectively Z s ) by SL(V). Since Z is projective, is also projective. GIT gives 
that 9Jt| is an open subset of the projective scheme 9)1$. The restriction Z s — > 
to the stable part is a geometric quotient, i.e. the fibers are SL(V)-orbits, and hence 
the points of SDtf correspond to isomorphism classes of 5-stable tensors. 
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It only remains to show that 9Jt$ corepresents the functor M s . We will follow 
closely [H-L2, Proof of Main Theorem 0.1, p. 315] 

Let (Et , <Pt j ut j N) be a family of J-semistable tensors (cf. (1.9)) parametrized 
by a scheme T. Then V := it t>i: (Et <8> ir x O x {rn)) is locally free on T. The family 
Et gives a map A : T — > Pic d (X), sending t € T to detE^. Cover T with small 
open sets Tj. For each i we can find an isomorphism 

(3 Tl ■ detE Ti — ► ~K*V 

(where V is the Poincare bundle in the definition of P at the beginning of section 
3), and a trivialization 

g Ti :V®0 Tl — » V| Tl - 
Using this trivialization we obtain a family of quotients parametrized by Tj 

g Ti : y (g) 7r^0 x (-m) -» E Ti , 

giving a map Tj — > And using the quotient and isomorphism we have 
another family of quotients parametrized by T 

(V^ s )® c ® (vr Ti ,(A~>® 6 u^fD ® 7r^O x ( S m))) V -» TV 

Then, using the representability properties of Ti and P, we obtain a morphism to 
TixP, and by lemma 3.1 this morphism factors through Z' and since a (5-semistable 
tensor gives a GIT-semistable point (theorem 3.6), the image is in Z ss . Composing 
with the geometric quotient to TXs we obtain maps 

fi-.Ti-fu z ss — an. 

The morphism f\ is independent of the choice of isomorphism (3^- A different 
choice of isomorphism gx t will change /j to /ij • /j, where /ij : T — > GL(V), so /j is 
independent of the choice of gx t ■ Then the morphisms /j glue to give a morphism 

/ : T — 0% 
and hence we have a natural transformation 

Recall there is a tautological family (3.6) of tensors parametrized by Z'. By restric- 
tion to Z ss , we obtain a tautological family of (5-semistable tensors parametrized by 
Z ss . If Ais —> Y. is another natural transformation, this tautological family defines 
a map Z ss — > Y, this factors through the quotient 971,5, and it is easy to see that 
this proves that 971,5 corepresents the functor Ms- 

Note that in [H-L2], the moduli space of stable framed modules is a fine mod- 
uli space. In our situation this is not true in general, because the analog of the 
uniqueness result of [H-L2, lemma 1.6] does not hold in general for tensors. □ 

Now we will give a criterion for S-equivalence. This is very similar to the criterion 
given in [G-S] for conic bundles. If (E, <p, u) and (F, ip, u) are two <5-stable tensors 
then we have seen that they correspond to the same point in the moduli space if and 
only if they are isomorphic. But if they are strictly 5-semistable, it could happen 
that they are S-equivalent (i.e. they correspond to the same point in the moduli 
space), even if they are not isomorphic. Given a tensor (E,{p,u), we will construct 
a canonical representative of its equivalence class (E s ,(p s ,u), hence (E,ip,u) will 
be S-equivalent to (F,ip,u) if and only if (E s ,tp s ,u) is isomorphic to (F s ,ip s ,u). 
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Let (E,<p,u) be strictly <5-semistable, and let (E,,m,) be an admissible weighted 
nitration, i.e. such that 

t 

(j2 m i( rP Ei ~ riP)) + p(<p,E.,m.)5 = 

i=l 

Let To be the set of pairs (k, I) where 1 < k < c is an integer, and I = (ii,...,i s ) 
is a multi-index with 1 < ij < t + 1, such that the restriction of p 

fe— 1 C — fc 

: . . . (B(E h (g) . . . E is ) e . . . e — ► (det E)® b ® D u 
is nonzero, and 

7r n H h7r ls = fJ,((p,E.,m.). 

If (A;, I) £ Zq and i 7 = (i' 1: . . . , is a multi-index with I' ^ I and ij < ij for all j, 
then 

(4.1) ^ fc)J , = 0, 

by definition of (i(ip, E,, m m ). Hence, if (k,I) € 1q, the restriction ip^j defines a 
homomorphism in the quotient 

fe— 1 c— fe 

(p' kjI ■■ o e . . . e o®(E' h ® . . . ® e o¥^¥o — > (det^)® b r» u , 

where i^' = Ei/Ei + \. If (fe, I) ^ 2o, then define ^ 1 = 0. Finally, we define 

e' = e[ e . . . e E' t+1 , = ¥/ fc)J . 

(fe,/) 

In the definition of 99' we are using the fact that det E = det E' , hence (E',ip',u) 
is well-defined up to isomorphism, and it is called the admissible deformation as- 
sociated to the admissible filtration (E,,m 9 ) of E. Note that it depends on the 
admissible weighted filtration chosen. 

Proposition 4.1. The tensor {E' , tp', u) is strictly 5-semistable, and it is S-equivalent 
to (E, ip, u) . If we repeat this process, after a finite number of iterations the process 
will stop, i.e. we will obtain tensors isomorphic to each other. We call this tensor 
(E s ,<p s ,u) 

(1) The isomorphism class of (E ,ip ,u) is independent of the choices made, 
i.e. the weighted filtrations chosen. 

(2) Two tensors (E, p, u) and (F, tp, u) are S-equivalent if and only if (E s , p s , u) 
is isomorphic to (F s , if) S , u) . 

Proof. We start with a general observation about GIT quotients. Let Z be a projec- 
tive variety with an action of a group G linearized on an ample line bundle O^(l). 
Two points in the open subset Z ss of semistable points are GIT-equivalent (they are 
mapped to the same point in the moduli space) if there is a common closed orbit 
in the closures (in Z ss ) of their orbits. Let z G Z ss . Let B{z) be the unique closed 
orbit in the closure G ■ z in Z ss of its orbit G ■ z. Assume that z is not in B(z). 
There exists a one-parameter subgroup A such that the limit zq = lim^o \ (t) ■ z 
is in G ■ z \ G ■ z (for instance, we can take the one-parameter subgroup given by 
[Si, Lemma 1.25]) . Note that we must have p,(z, A) = (otherwise zq would be 
unstable). Conversely, if A is a one-parameter subgroup with p(z, A) = 0, then the 
limit is GIT-semistable ([G-S, Prop. 2.14]). Note that G ■ z C G ■ z\G ■ z, and 
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then dimG • zq < dimG • z. Repeating this process with zo we then get a sequence 
of points that eventually stops and gives z € B(z). Two points z\ and Z2 will then 
be GIT-equivalent if and only if B(z\) = B(z2). 

Let (E,(p,u) be a 5-semistable tensor with an isomorphism / : V = H°(E(m)), 
and let z = (q, [<!>]) £ Z be the corresponding GIT-semistable point. Recall from 
proposition 3.7 that there is a bijection between one-parameter subgroups of SL(V) 
with /j,(z, A) = on the one hand, and weighted nitrations (E,,m,) of E with 
t 

(j2 m i( rP Ei-nP)) +fi{<p,E.,m.)8 = 

i=i 

together with a splitting of the filtration H°(E,(m)) of V = H°(E(m)) on the other 
hand. 

The action of A on the point z defines a morphism C* — ► R3 that extends to 

h:T = C — > Z, 

with h(t) = A(t) • z for t 7^ and /i(0) = lim t ^o ^(t) ■ z = zq. 

Pulling back the universal family parametrized by Z by h we obtain the family 

(q T ,E T ,(p T ,u) 

E T = E n ® t n c E ®c t _JV C[t] C £ ® c C[t, t^ 1 ] 

n 

q T : V <g> O x {-m) ® C[t] -^U © n F n ® O x (-m) <g> t n — ► £ T 

^ <g) 1 , — ► ' u n (g> t n 1 — > g(v n ) <8> t n 

V? T : (£ T ® S )® C — » (det St) 06 ® ® vr^iV 

fe— 1 c— A; fe— 1 c— fc 

Then, as in [H-L2, §4.4], (q t , E t ,(p t ,u) corresponds to (in particular, if t 7^ 0, 
then (E t ,<p t ,u) is canonically isomorphic to (E,<p,u)), and (Eo,(po,u) is the admis- 
sible deformation associated to (E,,m,). 

□ 

5. Orthogonal and symplectic sheaves 

In this section we apply the general theory of tensors to construct the moduli 
space of semistable orthogonal and symplectic sheaves. The only difference between 
these is whether the bilinear form is symmetric or skewsymmetric, hence we will 
first consider the orthogonal case, and at the end of the section we will add some 
comments about the symplectic case. We fix D u to be Ox (i-e. R is one point and 
T> is Oxxu, and hence we can drop u from the notation of tensors). 

Definition 5.1. An orthogonal sheaf is a tensor 

(E,ip) , if : E E — » Ox 

such that 

• (OS1) (det£)® 2 O x 

• ( OS2) ip is symmetric 

• ( OSS) E is torsion free 

• (OS4) V 9 induces an isomorphism E\ v — ► E\y on the open subset U where 
E is locally free. 
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An isomorphism of orthogonal sheaves is an isomorphism as tensors. 

It is easy to see that, assuming (OS1) and (0S3), the last condition is equivalent 

to 

• ( OS4') The induced homomorphism det E — > det E v is nonzero (hence an 
isomorphism). 

The following lemma justifies this definition for orthogonal sheaves. 

Lemma 5.2. There is a bijection between the set of isomorphism classes of or- 
thogonal sheaves with E locally free and the set of isomorphism classes of principal 
0(r)-bundles. 

Proof. The category of principal 0(r)-bundles is equivalent to the category whose 
objects are pairs (P, a) (where it : P — ► X is a principal GL(r)-bundle, a is a section 
of the associated fiber bundle P x GL(r) GL(r)/0(r)) and whose isomorphisms are 
isomorphisms / : P —> P' of principal bundles respecting a (i.e. it' o / = it and 
(/ x ido( r )) <t = a'). Note that this notion of isomorphism is not the same as 
isomorphism of reductions. 

The category of principal GL(r)-bundles is equivalent to the category of vector 
bundles of rank r. The quotient GL(r)/0(r) is the set of invertible symmetric 
matrices (send A G GL(r) to (J'A^ 1 A^ 1 )). Hence, a section a is the same thing as 
a homomorphism <p as in (OS2). Now it is easy to check that there is a bijection 
betweeen these sets of isomorphisms classes. 

□ 

Remark 5.3. Note that the categories are not equivalent: for example, let P be 
a simple principal G-bundle, i.e. the set of automorphisms of P is the center of G 
(a finite group), but the set of automorphisms of the corresponding G-sheaf is C*. 
We will have an equivalence of categories if we consider only isomorphisms (/, a) 
with a = 1, as in remark 1.2. This would be important if we wanted to construct 
the moduli stack, but since we are interested in the moduli space this is irrelevant, 
because the moduli space does not detect the group of automorphisms. 

Let (E, ip) be an orthogonal (or symplectic) sheaf. A subsheaf of F of E is called 
isotropic if <p\f®f = 0. Given a subsheaf i : F <^-» E, using the bilinear form ip we 
can associate the perpendicular subsheaf 

F 1 - = kei(E E v F v ), 
where Tp : E — > E v is the homomorphism induced by ip. 

Definition 5.4 (Stability). An orthogonal sheaf (E,p) is (semi)stable if for all 
isotropic subsheaves F C E, 

(5.1) P F + P F ^{<)P. 

An orthogonal sheaf (E, ip) is slope- (semi) stable if for all isotropic subsheaves F C E, 

deg(F) (<) 0. 

As usual, we can assume that F is saturated. A family of semistable orthogonal 
sheaves parametrized by T is a family of tensors 

(5.2) (E T , <pt, N), ip T : E T <£>E T — > tt t N, 

such that (det Et)® 2 is isomorphic to the pullback of some line bundle on T, ip F 
is symmetric, and ipx induces an isomorphism Et\u —* Ej< ® ^t^\u 011 the open 
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set U where Er is locally free, and such that the restriction to X x t for all closed 
points t is a semistable orthogonal sheaf. 

Using this notion of family, we define the functor Mo( r ) of semistable orthogonal 
sheaves. We will construct a moduli space corepresenting this functor (theorem 5.9). 

In proposition 5.7 we show that an orthogonal (or symplectic) sheaf (E, if) is 
(semi)stable in this sense if and only if it is e)-(semi)stable as a tensor (definition 
1.3), provided that 5\ > 0. Hence, the moduli space of semistable orthogonal (or 
symplectic) sheaves is a subscheme of the moduli space of (5-semistable tensors. In 
theorem 5.9 we show that it is in fact projective. We can also ask about slope- 
semist ability, and in proposition 5.8 we show that slope-(semi)stability in this sense 
and slope-T-(semi)stability as a tensor coincide if r > 0. If 5i = 0, then the notion of 
(5-semistability as a tensor is not equivalent to semistability as an orthogonal sheaf. 
At the end of the section we give an example of this. 

We start with some preliminaries. The intersection FCiF^ is an isotropic subsheaf 
of F. The following lemma gives exact sequences relating these subsheaves. 

Lemma 5.5. With the previous notation: 

(1) Let U be the open set where F, E and E/F are locally free. There is an 
exact sequence on U 

— ► F^u — > E\u — ► F v \u — ► 0, 

and hence rk(F ± ) = rk(E) — rk(F). If furthermore F is saturated (i.e. E/F 
is torsion free), then codim(X — U) > 2 and hence deg(F ± ) = deg(F). 

(2) If F is saturated, then F n F 1 - is also saturated. 

(3) There is an exact sequence 

(5.3) — > F<T\F ± — ► F © F ± — > F + F 1 - — > 

(4) F+F 1 - C {FHF^, rk(F+F x ) = rk((FnF ± ) ± ) ; and hence deg(F+F x ) < 
deg((FHF ± ) ± ). 

(5) Let F be a saturated subsheaf. If F n F ± / 0, then 

(5.4) deg(F) < deg(FnF x ), 
and ifFr\F ± = 0, then 

(5.5) deg(F) < 0. 

Proof. Since E/F\u is locally free, the last term in the following exact sequence is 
zero 

— > {E/F) y \u — > E v \u ?H F v \u — > Ext 1 ((E / F)\u , Ou) = 0, 
and hence i v \u is surjective. Combining this with (OS4) we get the exact sequence 

— > F% — » E\ v = E^\u ?H F^u — » 0. 

If E/F is torsion free, then codim(X — U) > 2 and we can use this sequence to 
obtain deg(F^) = deg(F). 

To prove item 2, first we show that F ± is saturated. The composition E — > E v — > 
F v factors as 

E -» E/F 1 - ^ F v . 
The sheaf F v is torsion free, and hence also E/F 1 - is torsion free. 

We conclude by showing that the stalk (E / (F n i 71 ^))^ = E X /(F X n F^f ) is torsion 
free for all points x G X. Let i> <G E x and let / / G rria; be a nonzero element in 
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the maximal ideal of the local ring of x, such that fv £ F x n F x . Since fv G F x , 
and F x is saturated, then v € F x . The same argument applies to F x , and hence 
v E F x f] F x . 

Items 3 and 4 are easy to check. To show item 5, if F n F 1 - ^ 0, use the exact 
sequence (5.3), together with items 1, 2 and 4. H FnF 1 - = 0, then = 
is a subsheaf of E of rank r, then deg(F) + deg(F^) < 0, and hence deg(F) < 0. 

□ 

The fact that on a generic fiber the quadratic form is nondegenerate has the 
following useful consequence: 

Lemma 5.6. // (E, ip) is an orthogonal or syplectic sheaf, then for all weighted 
titrations 

(5.6) ^{p,E.,m.) <0. 

Proof. First we will show that if Q : W ®W ^ C is & bilinear nondegenerate form 
on a vector space W, then Q € P(IF V <8> W y ) is GIT-semistable under the natural 
action of SL(T / F) (with the natural linearization induced on 0(1)). The point Q 
is unstable if and only if there is a one-parameter subgroup A of SL(W) such that 
lim^o A(i) ■ Q = 0. But this is impossible because det(A(t)) = 1, and then 

det(A(t) • Q) = det(A(t) Q T \{t)) = det(Q) + 0, 

hence Q is semistable. Then, using this and condition (OS4), it follows that 

fj,(<p,E.,m») < 

for all weighted nitrations. □ 

Proposition 5.7. Assume 5\ > 0. An orthogonal sheaf (E,ip) is (semi)stable if 
and only if it is S- (semi) stable as a tensor. 

Proof. To see that 5- (semi) stable as a tensor implies (semi)stable as an orthogonal 
sheaf, we apply the stability condition to the weighted filtration F C F^ C E with 
weights mi = m 2 = 1- By lemma 5.5(1), r = rk(F) + rk(F _L ). Since F is isotropic, 
fi((p, E,,m 9 ) = 0, hence the stability condition (1.6) gives the result: 

r(P F + P F± -P) = (rP F - rk(F)P) + (rP F± - rk(F ± )P) (^) 0. 

Now we will show that if (E, ip) is (semi)stable as an orthogonal sheaf, then it is 
8- (semi) stable as a tensor. We start with a vector space W and a nondegenerate 
bilinear form Q : W <g> W — > C. Let (W,, m.) be a weighted filtration with 

(5.7) n(Q,W„m.) = 0. 

Denote rj = dimWj. Take a basis of adapted to the filtration, and let A be 
the one-parameter subgroup of SL(W) associated to this basis and weights m,. Let 
7 = Yh=i rna^ as in (1.4). Since fi(Q, W.,m.) = 0, the limit Q' = lim t ^ A(t) • Q 
exists, and detQ' = detQ. Furthermore, we also have 

(5.8) (i(Q',W.,m.) = 0. 
Write Q and Q' as block matrices 

/ Qi,i Qi,2 ■ ■ ■ Qi,t+i \ 

Q2,l Q2,2 ■ ■ ■ Q2,t+1 



Q 



\ Qt+i,i Qt+1,2 ■ ■ ■ Qt+i,t+i / 
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Note that if + r y Tj < 0, then Qij = because of (5.7). We have 

f , 7r, + Irj < 
(5-9) Q' itj = < Qi,j , 7r, + 7r, = 

,0 , j ri + 7r, > 

The weights j n + 7 rj strictly increase with both i and j. Assume / 0. Then, if 
(a, 6) ^ and either a < i, b < j , or a > i, b > j, we have b = 0. In matrix 

form: 

/0 \ 

Q^- 


\ ooo/ 

Since det Q' = det Q ^ 0, in each row of Q' there must be at least one nonzero block 
(and the same for columns). This, together with (5.10) implies 



(5.10) 



Of 



(5.11) 



Q' = 



( 


o . 


. 


Q'i,t+i 


\ 





o . 


• Q'2,t 










Q't,2 ■ 


. 







V Q't+i,i 


o . 


. 





/ 



with nonzero blocks in the second diagonal, and zero everywhere else. Since Q' is 
nondegenerate, these blocks give isomorphisms for all 1 < i < t + 1 

Qi,t+2-i ■ Wi+2-i/Wi+i-i Wi/Wi-u 

and a short calculation then gives ri = r — r t +i-i- This, together with (5.11), implies 
that 



(5.12) 



Wt = W< 



t+l-i- 



Finally (5.9) and (5.11) imply that j n + 7 rt+2 _ i = for all 1 < i < t + 1. Then, 
using this and the definition of 7, 

= (7r l+1 + 7r i+1 _ t ) - (7r, + 7r i+2 _ t ) = r(mj - mt+l-i). 

Let (E,,m,) be a weighted filtration with ^(y?, E,,m,) = 0. We can assume that 
all subsheaves Ei are saturated. Apply the previous argument to W = E\ x , the fiber 
over a point where E is locally free, and Q the bilinear form induced by (p on the 
fiber. We have (5.12), hence it follows that E^ D E t+l _ i . Furthermore, as we have 
just seen rrii = mt+i-i and r j = r — r t +\-i for all i. Hence we can write 

t 

XI (rP Ei - nP) = 
i=i 

[(t+l)/2] 

i=i 

K*+l)/2] 

^ Tmr(P Ei +P E ±-P)(l)0, 

1=1 

where the last inequality is given by (5.1). 
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Let (E,,m 9 ) be a weighted filtration with fi(ip, E,,m 9 ) ^ . By lemma 5.6, it is 
strictly negative. 

We claim that deg(£ , j) < for all i. Assume that this is not true. Then there is 
a saturated subsheaf F C E with deg(F) > 0. By lemma 5.5(5), N = F n F 1 - / 0, 
and < deg(F) < deg(iV). By lemma 5.5(2), N is saturated, and by lemma 5.5(1), 
deg(iV) = deg(N ± ). Consider the weighted filtration TV C A r± C E with weights 
mi = r >T'2 = 1- Since N is isotropic, /j,(<p,N C N- 1 -) = 0, and since deg(iV) = 
deg(iV^) > 0, this weighted filtration contradicts (5.1). 

Hence, using deg(-Ej) < together with 5\ > 0, 

t 

( s ^m i (rP Ei - nP)j + u((p,E.,m.)S = 

i=i 

((E^7^w)+^^' m -)^) tn " 1 + 0(tri " 2) * 
i=i ^ n >' 

□ 

Proposition 5.8. Assume r > 0. An orthogonal sheaf (E,ip) is slope- (semi) stable 
if and only if it is slope-T- (semi) stable as a tensor. 

Proof. The proof of proposition 5.7, replacing the Hilbert polynomials Pp, Pe^ P, -- 
by the degrees deg(F), deg(-Ej), d,... proves that (E,(p) is slope- r-(semi)stable if 
and only if for all isotropic subsheaves F C E, 

deg(F) +deg(F ± ) < deg(£). 

We can assume that F is saturated, hence deg(F) = deg(-F^) by lemma 5.5(1), and 
since deg(E) = 0, the result follows. □ 

Fix a polynomial P. Recall that A^o(r) is the functor of families of semistable 
orthogonal sheaves. Define 9Jto(r) to be the subscheme of the moduli space of <5- 
semistable tensors corresponding to orthogonal sheaves with Hilbert polynomial P. 
The notion of S-equivalence for orthogonal sheaves is the same that was described 
in proposition 4.1. 

Theorem 5.9. The scheme 9Jto(r) ^ s a coarse moduli space of S-equivalence classes 
of semistable orthogonal sheaves. There is an open subscheme 9JIq^ corresponding 
to semistable orthogonal bundles. 

Proof. The proof that 97To(r) corepresents the functor A4o(r) is completely analogous 
to the proof of theorem 1.8 (see section 4), so we will not repeat it. The subscheme 
9#o( r ) * s °P en because being locally free is an open condition. 

Now we will prove that this moduli space is projective. Conditions (OS1) and 
(OS2) are closed conditions, so they define a projective subscheme 9711,2 of the 
moduli space of (5-semistable tensors. The lemma will be proved by showing that 
9JTo( r ) = SDTi ; 2- If (E, ip) is (5-semistable then E is torsion free, so it only remains to 
check that if condition (OS4) does not hold, then (E, ip) is 5-unstable. 

Assume that the homomorphism det E — > det E v induced by (p is zero. Then the 
sheaf E 1 - defined as 

— E ± — E E y 
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is nonzero. Let C be the cokernel of Tp 

E^E y C ^ 0. 

Taking the dual of this sequence and restricting to the open subset U of X where 
E is locally free, we get 

n j. f? vv l — rr\ f^L f? v l 

U — > u \u — >E \u — E\u — > E \u 

By (OS2) we have Tp\ v = Tp\^, hence ker^l^) ^ ker(^), and then C v \u = E L \u, 
and since codim(X — U) > 2, deg(C) = — deg(E ± ). The exact sequence on U 

o — » ^it; — » — » e v |[/ — » cv — » o 

implies that deg(£ ;± ) = 0. Consider the weighted filtration C E 1 - C E, mi = 1. 
We have 

fi((p,E.,m.) > 0. 

Recall that r = <5i(n - 1)!. Then 

r deg( J E ± ) - rk(E ± ) deg( J E) + E.,m.)T = E.,m.)T > 0, 

and hence {E,ip) is slope- r-unstable (definition 1.5), and in particular, J-unstable. 

□ 

Remark 5.10. The same proof gives that if (E,(p) is a slope- r-semistable tensor 
with r > 0, satisfying conditions (OS1), (OS2) and (OS3), then condition (OS4) 
holds. 



Example. We will give an example showing that, if we do not require <5i to be 
positive, the notion of 5-stability as a tensor (definition 1.3) is different from the 
notion of stability as an orthogonal sheaf (definition 5.4). We will check this by 
showing an example of an orthogonal sheaf whose <5-stability really depends on 5. 

Let X = P 2 , let p 1: p 2 , p 3 be three different points in P 2 , and consider the ideal 
sheaves I\ = I Pi and I2 = Ip 2 up :i - Let 

/ / 1/2 

{E,<p)=\I 2 ®I 1 ®Ox,\ 1 1/2 
\ \ 1/2 1/2 1 

In particular, the first summand I 2 of E is isotropic, and I 2 = I 2 ® I\- Let 5 = 
5\t + 82 € Q[t] be a polynomial as in (1.2). 

Lemma 5.11. If 5\ = and < 82 < 3/2, then (E,ip) is 8-unstable as a tensor. If 
81 > 0, then (E,ip) is 5 -stable as a tensor. 

Proof. The first claim is proved by considering the filtration Ox C I2 © h ® Ox- If 
8± = 0, then this filtration does not satisfy (1.6), hence contradicts semistability. 

Now we will prove that if <5i > 0, then (E, ip) is <5-stable. Using proposition 5.7, 
we only have to study nitrations of the form 

E\ C E 2 = E^ C E, 

with rk(£i) = 1, rk(£ , 2) = 2 and E\ isotropic and saturated. Using the Riemann- 
Roch formula we have 

P El +Pe 2 ~P = -2c 2 (E 1 ) - 2c 2 (E 2 ) + 2c 2 (E), 

so we need to estimate the second Chern classes of E\ and E 2 . 
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The sheaf E 2 = E± is saturated (see the proof of lemma 5.5(2)). Define the 
torsion free rank one subsheaf J 

o —> e 2 —> i 2 e h © Ox {a -^ ] J — > o, 

where a, b and c are respectively elements of Hom(/2, J), Hom(7i, J) and ~Kom(Ox, J) 
We have deg(J) = 0, so J is the ideal sheaf of a zero-dimensional subscheme of P 2 . 
We distinguish several cases: 

• If c / 0, then J = O x , and c 2 (E) = 3. 

• If a ytz 0, b ^ 0, then again J = Ox, and 02(E) = 3. 

• If a = 0, b ^ and c = 0, then E2 does not contain a subsheaf £1 with 
£2 = E^~, hence this cannot happen. 

• If a ^ 0, b = and c = 0, then again £2 does not contain a subsheaf £1 
with E 2 C -E^ - , hence this cannot happen. 

So we conclude that c 2 (E 2 ) = 3. The sheaf E\ is a rank one subsheaf of I2 © h © Ox , 
hence c 2 (E\) > unless E\ is the third summand Ox, but this is not possible 
because the third summand is not isotropic. Putting everything together, 

P El +Pe 2 ~P = -2c 2 (E l ) - 2c 2 (E 2 ) + 202(E) < 0, 

hence (E, (p) is 5-stable by proposition 5.7. □ 

Remark 5.12. Note that (E, ip) is stable as an orthogonal sheaf, but E is Gieseker- 
unstable as a sheaf. 

On the other hand, an orthogonal sheaf (E, ip) is slope-semistable if and only 
if E is slope-semistable as a sheaf. Indeed, if F is a saturated subsheaf of E with 
deg(F) > 0, then lemma 5.5(5) shows that the isotropic subsheaf FnF 1 is nonzero 
and has positive degree, hence (E, p) is slope-unstable. 

To obtain symplectic sheaves instead of orthogonal sheaves, we only need to take 
ip skewsymmetric instead of symmetric. It follows that det E = Ox (recall that for 
orthogonal sheaves we only had (detE)® 2 = Ox 

There is a bijection between the set of isomorphism classes of symplectic bundles 
and principal Sp(r)-bundles. The proof is the same as with orthogonal bundles, 
noting that the quotient GL(r)/ Sp(r) is the set of invertible antisymmetric matrices 
(send A G GL(r) to ( T A~ 1 JA~ 1 ), where J is the matrix representing the standard 
symplectic structure of C r ). 

All the results for orthogonal sheaves hold for symplectic sheaves, and in particular 
there is a coarse moduli space of S-equivalence classes of semistable symplectic 
sheaves. 

6. Special orthogonal bundles 
Definition 6.1 (Special orthogonal sheaf). A special orthogonal sheaf is a triple 
(E, p:E®E — ► O x , ij:detE — ► O x ) 

such that 

• (SOS1) ip is an isomorphism. 

• (SOS 2) p is symmetric. 

• (SOS3) E is torsion free. 

• (SOS4) p> induces an isomorphism E\ v — > E\y on the open subset U where 
E is locally free. 
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• (SOS5) det(v?) = V 2 . More precisely, let ip' : E -» E v and i// : O x -» det £ v 
&e i/ie associated homomorphisms. Then we require det(y') = ip <8> ip' . 

An isomorphism of special orthogonal sheaves is a pair (/, A) suc/i £/taf 
is an isomorphism, A € C* and t/ie following diagrams commute 

f® 2 . dct / 
(6.1) E ® E ^E'®E' det£ ^det£' 



Ox - - O x Ox — ^* Ox 

It is easy to see that, assuming (SOS1) and (SOS3), condition (SOS4) is equivalent 

to 

• ( SOS/) The induced homomorphism det E — ► det E y is nonzero (hence an 
isomorphism). 

Condition (SOS5) is equivalent to 

• (SO S3 ). Let U be the open subset where E is torsion free. For all x £ U, 
fix a basis of the fiber E x of E on x, Using this basis (and the canonical 
identification O x = C), ip restricted to x gives a symmetric matrix p(x), and 
ip restricted to x gives a complex number ip(x). Then we require det(<p(x)) = 
i)(x) 2 . 

This definition of special orthogonal sheaf is justified by the following lemma. 

Lemma 6.2. There is a bijection between the set of isomorphism classes of special 
orthogonal sheaves with E locally free and the set of isomorphism classes of principal 
SO(r) -bundles. 

Proof. Let SO(r) act by multiplication on the right on GL(r), and consider the 
quotient GL(r)/SO(r). Let A £ GL(r), and let [A] be the class in GL(r)/SO(r). To 
this class we associate the pair ( T A~ 1 A~ 1 , dct(^l _1 )). This gives a bijection between 
the set GL(r)/SO(r) and the set of pairs (B,(3), where B is a symmetric invertible 
matrix and (3 is a nonzero complex number such that 

det 5 = /3 2 . 

Given a principal GL(r)-bundle P (or equivalently a vector bundle E), a reduc- 
tion of structure group to SO(r) is a section a of the associated bundle P x GL ( r ) 
GL(r)/SO(r), and then this is equivalent to a pair of homomorphisms (if,ip) as in 
definition 6.1. 

The rest of the proof is analogous to the proof of lemma 5.2. □ 

Definition 6.3 (Stability). A special orthogonal sheaf (E, <p, ip) is (semi) stable if 
the associated orthogonal sheaf (E,tp) is (semi)stable. 

A family of semistable special orthogonal sheaves parametrized by T is a tuple 
(Et , <Pt , V'T j N) such that (Et,<Pt,N) is a family of semistable orthogonal sheaves 
(cf. (5.2)), and tpT '■ det Et — > vr^L is an isomorphism, where L is a line bundle 
on T. Two families are isomorphic if there is a pair (/, A : M — > M') where 
/ : E T — > E' T is an isomorphism, M® 2 = N, M'® 2 = N', A is an isomorphism, and 
the relative versions of the diagrams (6.1) commute. In this section (theorem 6.7) 
we will construct the moduli space of semistable special orthogonal sheaves (with 
fixed Hilbert polynomial). 
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There is a map between isomorphism classes 

f „ . , , . , . 1 / f Orthogonal sheaves 1 

(Speeml orthogonal sheaves) — { such that det E ^ Qx \ 

(E,<p,iP) — (E, tp) 

This map will induce a morphism between the corresponding moduli spaces. 

Lemma 6.4. Let (E,tp) be an orthogonal sheaf such that detE = Ox- 

If E has an automorphism f such that f ® f = He®e dnd det f = — iddet e> then 
the preimage of (E, tp) under the map f consists of exactly one isomorphism class. 

If E does not have such an automorphism, then the preimage consists of ex- 
actly two distinct isomorphism classes, represented by two special orthogonal sheaves 
(E,tp,tp) and (E,tp,—tp), differing in the sign of the isomorphism tp. 

Proof. Property (SOS5) implies that to obtain the isomorphism tp we have to extract 
a square root, so we obtain two special orthogonal sheaves P = (E, ip, ip) and P' = 
(E, <p, —tp) mapping to the given orthogonal sheaf. It only remains to check if theses 
two objects are isomorphic or not. 

If there is an automorphism / : E — > E with the above properties, then (/, 1) is 
an isomorphism between P and P' . 

Conversely, assume that there is an isomorphism (/, A) between P and P'. Then 
/' = (1/A)/ is an automorphism of E with f <g> f = id and det f = — id. □ 

Corollary 6.5. If r is odd, there is a bisection between the set of isomorphism 
classes of special orthogonal sheaves and the set of isomorphism classes of orthogonal 
sheaves with det E = Ox ■ 

Proof. Apply lemma 6.4 to / = — id# (multiplication by —1). □ 

In particular, for r odd, the moduli space of (semi)stable special orthogonal 
sheaves consists of the components of the moduli space of (semi)stable orthogonal 
sheaves with trivial determinant. On the other hand, if r is even and E is simple, 
then for each orthogonal sheaf with trivial determinant, we have two nonisomorphic 
special orthogonal sheaves. From now on we will assume that r is even. 

Fix a Hilbert polynomial P. Let m be a large integer number as in section 3. Let 
V be a vector space of dimension P(m). Let (g, E, <p, tp) be a tuple where (E, tp, tp) 
is a semistable special orthogonal sheaf and g is an isomorphism between H°(E(m)) 
and V. As in section 3, the homomorphism ip gives a vector 

$ e {V m Y ®H Q {O x {2m)) 
We denote <& s = $® r / 2 the associated vector 
(6.2) § s G Sym r / 2 {(V® 2 ) y H°(O x (2m))) . 

The homomorphism tp induces a linear map 

* : l\' V — ' H°(det(E)(rm)) — ► H°(O x (rm)), 
and hence a vector (that we denote with the same letter) 

{J\ r Vy ®H°{O x {rm)). 

These two quotients give a point [3> s , VP] in the projective space P defined as 
p(Sym r / 2 ({V® 2 Y (8) H°{O x (2m))) {(/\ r V) v fl°(O x (rm)))). 
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It is easy to check that the point only depends on the isomorphism class of the 
tuple. Here it is crucial that we took the r/2-symmetric power in (6.2): take the 
isomorphism Aid : E — > E (multiplication by A 6 C*). It sends <I> to A 2( &, and 
<I> to A r $, hence it sends [^s,^] to [A r 3> s , A r \P], and this is the same point in the 
projective space. 

Let H be the Hilbert scheme of quotients as in section 3, and then given a tuple 
(g,E,ip,ij)) we associate a point (q, [$ s ,Vl/]) in H x P. The points obtained in this 
way have the following properties: the vector $ s is of the form $® r / 2 , $ factors as 

V® 2 ® O x (-2m) ^ H°(O x (2m)) <g> O x {-2m) 

E m 1 _ Ox , 

the homomorphism \I> factors as 

/\ r V® Ox(-rm) ^ H°(O x (rm)) ® O x {-rm) 

det E - O x , 

and det(<p) = if) 2 as in (SOS5). 

Let Z' be the closed subset ofHxP defined by these properties. Given a point 
z £ Z' we can recover the tuple up to isomorphism. Define the parameter space 
Z as the closure in Z' of those points obtained from semistable special orthogonal 
sheaves. 

Let 5 be a polynomial as in (1.2) and with <5i > 0. Define a polarization on Z by 

0^ ni ,n 2 ) :=p^O w (ni)®pJ,0 P (^), 

where n>2 is a multiple of r/2, ni is an integer, and 

ra 2 _ P(l)S(m) - 5{l)P{m) 
n\ P(m) — 2S(m) 

The projective scheme Z is preserved by the natural SL(V) action, and this action 
has a natural linearization on O^fai,^)- 

Proposition 6.6. A point (g, E, <p, tp) is GIT- (semi) stable if and only if the special 
orthogonal sheaf (E, <p,ip) is (semi)stable (definition 6.3). 

Proof. The parameter space Z for orthogonal sheaves is a subscheme of H x P, 
where 

P = F((V® 2 ) V <g> H°(O x (2m))) 

(this is a particular case of the parameter space defined in section 3). Let Oz(n±, 112) 
be the polarization defined in (3.7), and consider the natural linearization of the 
action of SL(V) on this polarization. There is a morphism 

Z -U Z 

{g,E,<p,ip) 1 — ► (g,E,ip) 

with f*Oz(ni,n 2 ) = 0^(ni,n 2 ). This morphism is equivariant with respect to 
SL(V), and the linearizations are compatible. Property (SOS5') implies that / is 
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finite etale (because Z is given locally by the equation det(ip(x)) = ip(x) 2 ), and then 
it follows that a point in Z is GIT-(semi)stable if and only if its image in Z is GIT- 
(semi)stable. The result follows from theorem 3.6, proposition 5.7, and definition 
6.3. □ 

Let -Mgo(r) be the functor of families of semistable special orthogonal sheaves. 
Let 9Kgo(r) be the GIT quotient of Z by SL(y). Let (E,ip,ip) be a semistable 
special orthogonal sheaf. Let E s and (p s be defined as in section 4. There is a 
natural isomorphism between detE s and det-E, then composing with ip we obtain 
an isomorphism ip s : det E s — > Ox ■ 

Let (E,(p,tjj) and (E', <//, ip') be two semistable special orthogonal sheaves. They 

o o o S S S 

are S'-equivalent if and only if (E°, ip ,ip ) is isomorphic to (E' ,ip' ). 

Theorem 6.7. The projective scheme 9JTgo( r ) the coarse moduli space of S- 
equivalence classes of semistable special orthogonal sheaves. There is an open subset 
9#go(r) corresponding to semistable special orthogonal bundles. 

The proof is completely analogous to the proof of theorem 1.8 (section 4). 

7. GL(r)-REPRESENTATIONAL PAIRS 

Once we have constructed the moduli space of tensors, it is easy to obtain moduli 
spaces for GL(r)-representational pairs. In the case of dim(X) = 1, this is done in 
[Sch], but since it does not depend on the dimension of the base X, the same argu- 
ments apply here. In [Ba], Banfield considered pairs (P, a), where P is a principal 
G-bundle (G any reductive group), and a is a section associated to P by a fixed rep- 
resentation p. He defined stability, and proved a Hitchin-Kobayashi correspondence. 
Now we will construct the moduli space, when G = GL(r). 

Fix a polynomial 5 as in (1.2). Let p : GL(r) — > GL(n) be a representation 
sending the center of GL(r) to the center of GL(n). Consider a triple 

(7.1) (E, if; : E p ^ D u , u), 

where E is a vector bundle of rank r on X, and E p is the vector bundle of rank n 
associated to E and p. Using [F-H, prop. 15.47], it can be shown that there exist 
integers s > 0, b, c > 0, and a vector bundle F such that 

(E® s f c ® {detE)~® b 9* E p ®F 

(see [Sch, cor 1.1.2] for details). Then a triple (7.1) is equivalent to a tensor (E, if, u) 
such that 

(7.2) ¥>If = 0, 

and we say that the triple is 5-(semi)stable if the corresponding tensor is. Since 
the condition (7.2) is closed, the moduli space of 5-semistable triples is a closed 
subscheme of 9JtJj, the open subscheme corresponding to tensors with E locally free. 
It is easy to check that the definition of stability in [Ba] coincides with our slope-r- 
stability. 

In [MR], Mundet generalized Banfield's work. He fixes a Kaehler manifold Y and 
an action p of a reductive group G on Y, and considers pairs (P,a), where a is a 
section of the associated fiber bundle P xqY. He defined stability and proved a 
Hitchin-Kobayashi correspondence. Now we will construct the moduli space, for the 
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case when G = GL(r), and Y is a projective (or more generally, quasi-projective) 
scheme. 

Consider an action of GL(r) on a projective scheme Y, together with a lineariza- 
tion of the action on an ample line bundle L on Y. Assume that the center of GL(r) 
acts trivially on Y. Consider a pair 

(7.3) (P, ff :I^Px GL(r) n 

where P is a principal GL(r) bundle on X, and a is a section of the fiber bundle 
associated to P with fiber Y. We fix the topology type of P and the homology 
class [<t(X)] of the image. Fix k large enough so that we have a natural embedding 
F ^ F(H°(F,L® k ) v ). Since the action of the center of GL(r) is trivial on Y, the 
induced representation 

p : GL(r) — > GL(H°(F, L® k )) 

sends the center of GL(r) to the center of GL(H°(F, L® fc )). Let E be the rank r 
vector bundle corresponding to P. Since we have fixed the topology type of P, the 
Hilbert polynomial Pe of E is also fixed. The section a gives a homomorphism 

ip : E p — ► D u , 

for some line bundle D u , whose degree a depends on the homology class [c(-X')] 
of the image. Take R = Pic a (X), and let T> be a Poincare bundle. We obtain 
that a pair (7.3) is equivalent to a triple (7.1) with the property that the section 
ip' : X -> F(E^) factors through P x GL(r) Y. We define a pair (7.3) to be 5- 
semistable if the corresponding triple is, and hence the moduli space of <5-semistable 
pairs (7.3) is a closed subscheme of £Ut§ . We can also take Y to be quasi-projective, 
and the moduli space will also be a subscheme (not necessarily closed) of 9Jt°. 
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